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ABSTRACT 

This study examined ways in which two independent 
variables, peer collaboration and the use of a specific tool (the 
TAPS interface), work together and individually to shape students' 
problem-solving processes. More specifically, the researchers were 
interested in determining how collaboration and TAPS use cause 
metacogni t ive processes to differ from those that occur when students 
engage in individual problem solving using standard tools. The TAPS 
systems help students develop metacogni t ive skills and awareness. 
Forty-eight undergraduate students were selected and randomly 
assigned to one of four research groups: pairs and TAPS; individuals 
and TAPS; pairs and standard tools; and individuals while thinking 
aloud. Protocols c£ these problem-solving sessions were videotaped 
for analysis. For coding and analysis, each protocol was broken down 
into 15-se-cond segments. Result* showed that TAPS users differed from 
users of conventional tools in ^oth time taken in problem solving and 
reading time; TAPS use appeared to serve as a catalyst for more 
strategic planning behavior; collaboration also produced more 
planning behavior; and more behavior associated with met acogni t ive 
monitoring occurred in the pair groups. (Contains 16 references.) 
(JLB) 
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Objectives of Study 



The main purpose of this study was to examine ways in which two independent 
variables, namely (1) peer collaboration and (2) use of a specific computer tool (the TAPS 
interface), work together and individually to shape students' problem-solving processes. 
Because the TAPS interface was designed to enhance metacognition, we were especially 
interested in determining how collaboration and TAPS tool use cause metacognitive 
processes to differ from those that occur when students engage in individual problem 
solving and use standard (pencil, paper and a calculator ) tools. This question is motivated 
by the work of Vygotsky (1978), Lave and Wenger (1991), and others, who have 
postulated that thinking is both physically and socially situated, meaning that problem tasks 
can be significantly shaped and changed by the tools made available and the social 
interactions that take place during problem solving. Like other researchers (e.g., Newman, 
Griffin, & Cole, 1989), we seek to understand whether and how the "same" problem- 
solving task is transformed by tool use and different social organizations. This question is 
of general interest to the extent that peer collaboration on word problems is common 
instructional practice and that the TAPS interface resembles a number of non modeling, 
reflection-type instructional computer tools that have been developed in recent years (e.g., 
Collins & Brown, 1988; Lajoie & Deny, 1993). 

In addition to this basic research question, this study also addresses a practical 
issue: a need to obtain baseline data on how cognitive processing is shaped by an 
instructional system's interface alone; that is, the basic interface design unenhanced by 
tutoring or other instructional treatment. This baseline data will help us n: ore precisely 
gauge the 'value added 1 by various intelligent tutoring strategies that will overlay the 
interface in future studies. 



In previous research (e.g., Deny, Hawkes, & Kegelmann, 1991) we have 
informally observed that students who engage in collaborative problem solving using 
TAPS interface tools appear to exhibit what most cognitive psychologists would regard as 
more mature metacognitive behavior than do students who think aloud while solving 
problems alone using pencil and paper. That is, student pairs working together on TAPS 
appear to reflect upon their problem-solving processes more often, conduct more strategic 
planning, and overtly monitor and check their work more frequently. Overall, these 
observations suggest that even without tutoring or other instruction, social problem solving 
using TAPS interface tools may help shape and develop students' thinking processes in 
potentially productive ways. This expectation formed the basis for the research hypotheses 
of this study. 

The primary questions addressed by our study can be summarized as follows: 
Compared to students who solve problems using the TAPS interface, will students who 
* v-* solve problems without using TAPS tools exhibk different types and frequency of 

metacognitive activity? Also, when students are asked to think out loud while working 
individually, do they exhibit metacognitive perfomiance that is different from what is 
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exhibited in collaborative problem solving? Finally, do the effects of computer tool use 
differ for pair versus individualized instructional conditions? One of our hypotheses was 
based on the assumption that by making covert, abstract reasoning processes visible, public 
and manipulable, TAPS would serve as a necessary catalyst for reflective metacognitive 
activity that would not take place in its absence (Vygotsky, 1978; Collins & Brown, 1988). 
Another hypothesis involved eliminating the possibility that the relevant key to enhancing 
metacognitive activity during problem solving is merely thinking aloud, for thinking aloud 
itself can produce more reflection and self-monitoring and has been associated with 
improvements in problem solving performance (e.g., Gagne' & Smith, 1962). 



Background : Description of System 

TAPS is a computer-based instructional tool, programmed for the NeXT machine 
(soon to be available for 486 machines running Microsoft Windows) and designed to help 
students develop metacognitive skills and awareness, where metacognition refers to one's 
knowledge of the thinking process itself and how to control it. Examples of metacognitive 
processes targeted by TAPS instructional activity include planning, self-monitoring, and 
self-awareness of the nature of complex problem solving and one's thoughts and behaviors 
during problem solving. A metacognitive approach is justified by numerous detailed 
analyses of typical problem-solving difficulties experienced by older children and adults 
(Deny, 1989; Schoenfeld, 1983), which have shown that many problem-solving errors are 
due to metacognitive failure rather than, or sometimes in addition to, lack of basic 
mathematics knowledge. 

In designing TAPS we embraced Vygotsky 's (1978) notion of cognitive tools - 
objects provided by the learning environment that permit students to incorporate new 
auxiliary methods or symbols, which otherwise would be unavailable, into their problem- 
solving activity. The major cognitive tool provided by TAPS is a graphics interface that 
facilitates construction of problem trees, network structures showing interrelationships 
among all relevant sets in a problem situation, specifying the subgoal structure of the 
problem, and illustrating a solution path. Such trees provide graphic reifications of abstract 
structures that otherwise would be invisible to students, making them public and available 
for discussion, manipulation, and comparison. This type of instructional environment is 
widely believed to encourage and afford greater metacognitive awareness (e.g., Reusser, in 
press; Collins & Brown, 1988). 



Insert Figure 1 About Here 



As shown in Figure 1, a student or group working on TAPS is presented with a 
complex word problem to solve. (Henceforth in this discussion, the term "student" should 
be understood to mean either an individual student or a group of students working together 
on the system.) Complexity and story content of each problem can be varied to match the 
abil> "s and backgrounds of the student. To make possible the construction of a tree, the 
ir idee provides a menu of blank three-node subtree diagrams ("schemas"), shown to the 
rhmt of the screen in Figure 1. Students proceed by selecting subtree diagrams from this 
rr nu and moving them into the workspace one at a time. 

In the workspace each subtree is filled in with labels, values and an operator so that 
it represents a particular set relationship from the problem. When filled in, each subtree 
represents a simple equation such that the two nodes at the top of the subtree represent two 
sets that can be related by an arithmetic operation to produce the set value represented by 
the bottom node. It is important to note, however, that in the naming of sets, students 



make explicit their semantic understanding of the concepts underlying arithmetic 
operations. By requiring this set labeling and providing possibilities, the system 
unobtrusively scaffolds students in the process of thinking conceptually about arithmetic 
structures. 

Each blank schema in the subtree menu is shaded differently to indicate where it can 
be attached to another subtree. System users select the appropriate subtree diagram and 
then "click" with the mouse at the point on the problem tree (in the workspace) where the 
new schema will be attached. In this manner, students can chain subtree schemas together 
to form complex tree structures, as shown in Figure 1 . 



Method and Data Source 

Volunteer subjects, who received course credit for participation, were solicited from 
a large undergraduate educational psychology course. All volunteers completed a survey 
instrument designed to help determine their attitudes and abilities toward mathematics. 
Since TAPS is designed primarily for lower math achievers, students were eliminated if 
they reported evidence of above-average achievement in college mathematics. Examples of 
the type of evidence considered include 1 the following: Above-average performance on 
SAT or ACT mathematics achievement tests; advanced college courses in mathematics with 
a grade of B or better; A grade of "A" in a recent college-algebra course; Self-ratings on 
attitude scales such as, "I am very good at word problems" or "I really hate math word 
problems." 

Forty-eight students were selected and randomly assigned to one of four treatment 
conditions. In Treatment 1, eight pairs of students work collaboratively in pairs on the 
TAPS system. Each session consisted of presenting the students with one training and one 
practice problem on the system, followed by two experimental problems, the "pizza 
problem" and the "politics problem," shown in Table 1. The experimental problems were 
presented in counterbalanced order within treatment, so that four pairs received the pizza 
problem first, and four received the politics problem first. 



Insert Table 1 About Here 



Treatment 2 was identical to Treatment 1, except that eight students worked their 
problems on TAPS individually rather than collaboratively, and were instructed to think 
aloud du;:ng problem solving. In Treatment 3, student pairs worked collaboratively on 
identical problems, but without the TAPS system. Rather, subjects in Treatment 3 used 
pencil a r id paper for developing their joint solutions. In treatment 4, eight students worked 
independently using pencil and paper while thinking out loud. 

Protocols of these problem-solving sessions were videotaped for analysis. A 
practice effect was possible with TAPS subjects, who appeared to use the system with 
greater ease for their second experimental problem. Thus, to minimize the effects of 
unfamiliarity with the TAPS system, we analyzed only the second experimental problem 
from each protocol. 

For coding and analysis, each protocol was broken down into fifteen-second 
segments. Coding involved studying each fifteen-second interval on each tape and 
assigning codes to indicate the type of behavior observed during each interval. Coding 
followed the scheme shown in Table 2. This coding system was derived over a period of 
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several months through the research and discussions of videotapes carried out in the first 
author's graduate research group. It represents a synthesis of ideas from the literature on 
metacognition and social problem solving (e.g., Schoenfeld, 1987; Clements & Nastasi, 
1988). 



Insert Table 2 About Here 



During development of the coding system we examined the reliability of an 
intermediate version by having two researchers independently code the same three 
videotapes. A reliability coefficient was derived by computing the average percentage of 
matching fifteen-second coding intervals for each general coding category. These initial 
reliability computations ranged from a high of .93 for the category of reading 
to a low of .70 for the category of monitoring. At this point the two coders met to discuss 
their discrepancies and to clarify and elaborate the coding system for the purpose of 
obtaining better reliability. After two rounds of reliability checks and discussions, all tapes 
were coded by a single rater. The reliability of this final coding was estimated by having a 
second researcher code a random sample of four protocols and then calculating the 
percentage of agreement between the two raters. All reliability computations in this final 
analysis were greater than .90. 



Results 

To help del ermine how different social and tool conditions differentially affected 
cognitive processing, the four treatments were compared on the basis of total time spent in 
problem-solving activity (expressed as average number of fifteen-second intervals used to 
complete the experimental tasks in each condition), and also on the basis of what types of 
cognitive activity were observed during problem solving in each treatment (expressed as 
percentage of intervals spent in each of the coding categories of Table 2). Results of this 
comparative analysis for each major coding category are displayed in Tables 3 and 4 and 
are graphically depicted in Figures 2 through 7.2 Since our analyses revealed no 
statistically significant differences between the pizza and Macintosh problems, all data and 
analyses reported combine the two problems. 



Insert Tables 3 and 4 and Figures 2-7 About Here 



Table 3 shows the mean "duration" of each treatment as well as the percentage of 
total problem solving time for each treatment spent in activities related to reading, planning, 
monitoring, attending to the computer interface, or actually carrying out a solution. The 
coding category exploration is not shown in Table 3 because no exploratory behavior was 
observed during processing of the experimental problem, although some exploratory 
behavior did appear during practice sessions. We note that the sum of total percentages for 
each treatment exceeds 100% because the coding categories are not mutually exclusive. For 
example, it is very possible to be engaged in planning a strategy and at virtually the same 
time be monitoring how well that strategy is working. 

Statistical tests for main effects (pen/paper versus TAPS and pairs versus 
individuals) for the categories of reading, planning, monitoring, and carrying out a solution 
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strategy are shown in Table 4. These tests were computed using adjusted Mann-Whitney 
U tests, a procedure that does not require homogeneity of variance. It was also possible to 
test for significance of interactions by comparing the significance of the difference between 
two U statistics, provided there is homogeneity of variance on at least one of the two main 
effects involved (R. Serlin, personal communication, April, 1993). Homogeneity tests 
were computed using Levene's test, resulting in the rejection of the homogeneity 
hypothesis for only one main effect, the individual versus pair comparison in the coding 
category of planning. Hence, it was possible to test for all possible interactions. 

As indicated in Table 3 and Figure 2, students working with the TAPS tool spent 
nearly four times longer working on their experimental problem compared to subjects in the 
non-TAPS conditions, a clearly significant difference. It is possible to adjust these means 
by removing time spent attending to the computer interface, on the assumption that the 
additional time added by the interface is possibly irrelevant since it presumably would 
disappear with further practice. This adjustment is reflected in Figure 3, where bars labeled 
"TAPs problem time" reflect time spent in problem-solving activity with interface time 
removed. Using either adjusted or unadjusted means, however, the time difference is 
marked and obvious: Without guidance, TAPS interface users require much more problem 
solving time than do subjects solving the same problems with the standard pencil and paper 
tools. 

Given this difference, it is natural to ask whether TAPS subjects performed 
noticeably better or worse than subjects using pen and paper. Our current sample size was 
too small to answer this question for certain, but the probability of obtaining a correct 
solution was .50 (4 out of 8 protc jols) for Pen/Paper-Individual, .50 (4 out of 8 
protocols) for Pen/Paper-Pair, .875 (7 out of 8 protocols) for TAPS-Pair, and .750 (6 out 
of 8 protocols) for the TAPS -Individual group. This difference is not statistically reliable, 
although it is suggestive of a positive trend. 

Several a priori hypotheses were advanced with respect to the metacognitiye 
categories of planning and monitoring. It was hypothesized that both social interaction and 
use of the TAPS interface would increase attention to planning. As shown in Figure 4, 
pairs using the TAPS system spent a much greater percentage of their time in planning 
activity than did any other group. Pairs did significantly more planning than did 
individuals (z=2.47, p<.05), and TAPS subjects did significantly more planning than did 
subjects using pen and paper (z=1.97, p<.05). The test for the interaction was not 
statistically significant (z = -0.544), although TAPS tool use combined with collaboration 
produced substantially greater planning activity than did any other condition. Further 
inspection of data indicated that this was predominately forward, short-range planning of 
strategy. 

In addition, it was expected that pairs working together would display relatively 
more monitoring behavior as a natural consequence of cooperative engagement in problem 
solving. It was also anticipated that TAPS would increase monitoring behavior, which 
would result from making abstract aspects of problem solutions more concrete and visible, 
thus prompting more discussion and questioning. As shown in Table 4, pairs did 
significantly more monitoring than did individuals (z = 4.35, p< .05), but the difference 
between TAPS and non-TAPS subjects was not statistically reliable (z = 0.31). 

Post hoc tests for the categories of reading and carry-out also were conducted, 
revealing one statistically significant main effect: Those using the TAPS system spent only 
30.34% of their problem-solving time in the activity of reading, whereas subjects using 
standard tools (pen and pape. , spent 69.62 percent of their time reading. Closer inspection 
of coded data revealed that not only did percentage of reading time differ, but that TAPS 



use greatly changed the nature of the reading activity. Students using pen and paper were 
able to take notes as they read and to underline or mark their text to facilitate focusing. But 
more evidently, students using traditional tools seemed to interleave reading with other 
problem solving processes so tightly that it was not easy, for the traditional tools 
conditions, to isolate reading from other problem solving processes such as carrying out a 
solution. For TAPS subjects, reading was a distinctive form of processing that was more 
easily separated from activities such as carrying out and planning. 



Discussion 

Current findings add validity to the situated cognition viewpoint (e.g., Newman, 
Griffin, & Cole, 1989) that the same problem-solving task under different social 
organization and tool conditions is not the same task at all. This study has added to ovjr 
understanding of how word problem tasks are changed by two common situational 
modifications: use of a computer interface and the social condition of pair problem solving. 
Several differences related to tool use were evident: Although they solved identical word 
problems, TAPS interface users differed from users of conventional tools in both time 
taken for problem solving and relative reading time. There also were clearly observable 
qualitative differences in reading style between TAPS and non-TAPS subjects. With 
conventional tools, but not with TAPS, reading and other problem solving activities are 
tightly intertwined such that reading is not easily identified as a distinctive processing stage. 
Moreover, there is a wider variety of reading activity, including note taking and 
underlining, with the conventional tools. Another difference was that TAPS use appeared 
to serve as a catalyst for more strategic planning behavior. 

With respect to the influences of social (versus individual) problem solving, 
collaboration also produced significantly more planning behavior, such that the combined 
use of TAPS and collaboration produced a marked increase in planning. This planning was 
mostly short-term planning of problem-solving strategy, such as how to approach the next 
step. Also, significantly more behavior associated with metacognitive monitoring occurred 
in the protocols for pairs. This finding relates to our research question regarding the 
relative effects of social negotiation versus thinking aloud. With respect to monitoring, at 
least, merely thinking aloud about problem solving does not produce as much 
metacognitive activity as does discussion between two cooperating problem solvers. We 
plan to investigate the effects of collaboration in greater depth through further analyses of 
the social interaction on our tapes. 

The search for instructional activities that enhance reflection and focus students' 
attention on the quality of their thinking processes, rather than procedural efficiency, is 
compatible with goals of the current reform movement in mathematics education (e.g. 
NCTM Standards, 1989). To the extent that more planning and monitoring are desirable 
byproducts of instructional activity, then pair problem solving with the TAPS tool does 
appear to encourage certain desirable behaviors, especially evident when the conditions of 
social problem solving and tool use are combined. Compared with conventional 
pencil/paper problem solving, TAPS problem solving is more like the separate stages of 
problem solving as described in models of "ideal" problem solving, such as the one 
proposed by Polya (1957). One can thus easily envision the use of TAPS with problem 
solving instruction or tutoring that builds on students' natural tendencies to plan, monitor, 
and work in discrete stages. The fact that the TAPS interface can be expected to slow 
problem solving so greatly might seem undesirable to some. However, we emphasize that 
this study has examined baseline processing changes associated with an interface design 
unaccompanied by tutoring, instruction, or other on-line help. It should be possible to 
increase the efficiency of TAPS use by adding improvements in interface design, as well as 



various forms of tutoring or other on-line assistance. 
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Table 1. Experimental Word Problems 



The gang is sending Sam out for pizza. Dominoes sells its large pizza with three toppings 
for 12 bucks. Extra cheese is 2 bucks, and extra sauce is another 2 bucks. Additional 
toppings are $J .00 each. Sam is supposed to get two large pizzas. They want one with 
four and one with five toppings. Both pizzas will have extra cheese. Sam has a coupon 
that gets him 2 dollars off per pizza for an order of two or more. So far he has collected 
20 dollars from the group, How much more must he collect? 



In the State of Forgottonia, Representative Black is pushing a new college loan program, 
fifty minority and 80 majority party representatives present today are in favor of the bill. 
One hundred representatives present today are unalterably opposed to the bill. 
Following Governor White's impassioned attack on the bill, enough uncommiwed votes 
switched to opposed so that the same number of representatives are pledged against the 
bill as are pledged in favor of the bill. No other vote switches tool place. If there are 
still 120 uncommitted representatives on this bill, how many were uncommitted before 
Governor White's speech? 
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Table 2. DESCRIPTION OF PROTOCOL CODING SCHEME 



R: READING 



A fifteen-segment interval was coded M R" if any reading behavior occurred during that 
interval. Reading is defined as any attention paid to the actual words of the problem. 
Something other than reading is taking place if any math work is performed, if there is any 
discussion of how the problem is to be solved, or even speculation as to whether or not the 
problem is understood correctly. In addition to the main category, reading was coded by 
the subcategories described below 

Sub-categories of Reading: 
R.r: Reading (First Time) 

First Time Reading is narrowly defined as the first time each word in the problem is 
read. Most subjects read the whole problem through once at the beginning, but 
occasionally a subject will work on part(s) of the problem before going back to read the 
rest. Note taking, underlining, focusing and inferencing can take place in parallel (if 
done as the words are read for the first time) but these are different activities. 



R.a: Reading(Again) 

Reading Again is the partner of first time reading. It includes any reading of the 
problem after the first reading. Note taking, underlining, focusing and inferencing can 
take place in parallel but these are different activities. If the subject goes back to 
underline (infer or focus) without reading the words aloud first, it is coded as only 
underlining (inferencing or focusing) and not as reading( again). 

R.u : Reading(Underline/Mark) 



Underlining or marking consists of simply identifying words in the problem 
statement in any way to make them visibly distinguishable from other words. If 
attention is drawn to particular words, with no actual underlining or marking, it should 
be coded as reading(focusing) instead. 

R.f: Reading(Focusing) 

Reading(focustng) consists of drawing attention i ok<*j in the problem statement. 
If the words are marked or underlined, it was coded as reading(underline/mark) 
instead. Focusing is often preceded by remarks like, 'it says here', 'look at this', or 
'oh, I see 1 which are used to bring attention to part of the problem statement. Focusing 
is actually a broader category than underlining/marking, distinguished in that the focus 
leaves no permanent record while underlining/marking does. 

R.n: Reading(Note Taking) 

Note taking consists of any copying of information from the problem statement to 
any other medium, such as scratch paper. TAPS does not currently have a facility for 
note taking on the computer. Writing is considered note taking only when it is 
information directly from the problem that is being written. Computations and writing 
of inferences were viewed as not note taking but as means of adding information. 
Jotting the problem information in a different form, order or representation than the 
original counts as note taking so long as no information is added. 



R.i: Reading(Inferencing) 



Inferencing consists of looking at the problem statement and directly inferring 
information that is not explicitly there. Calculating numbers not given but deducible is 
not inferencing. For example, if an allowance is $1 a week, it would be an inference to 
say there are 52 weeks in a year but a calculation to say the total allowance in a year is 
$52. Inferencing is often accompanied by remarks like, 'Oh, that means...', 'that is 
saying that...', or by simply stating as fact a statement not in evidence. Questioning or 
clarifying which results eventually in an assumption or shared assumption (e.g. Should 
we assume that . . . ?) was not be classified as an inference. An inference was viewed 
as a product of reading the problem statement and not a negotiated agreement. 

PL: PLANNING 

A fifteen-second interval was coded "PL" if it included decision-making about what to 
do. The four subcategories are range(how far away is the focus of the plan), direction (is 
the plan forward or backward), goal(has a goal been stated), and what(is it an oper ;ion, 
goal or strategy that is being planned). Normally a planning activity will be classified 
according to all four subcategories. It is possible, however, to discuss strategy, for 
example, without discussing other aspects of planning. It is also possible to plan interface 
use as opposed to planning 

the problem solution. Planning activity can be identified by phrases such as 'what we need 
to do'. Simply stating what is being done as it is being done is not planning (see carry 
out). 

Sub-categories of Planning 

PL.r: PIanning(Range: Immediate or Distant) 

Planning is considered immediate range if the focus is on the next thing to do and 
distant if the focus is beyond the next thing. Immediate planning is often identified as 
deciding 'what to do next' while distant planning is often identified as 'what needs to 
be done'. Immediate planning is usually followed by the planned action, but it is still 
immediate planning if the subject changes his or her mind, forgets, or is distracted. 

PL,d: Pianning(Direction: Forward or Backward) 

Planning is considered forward if it starts with 'lie givens of the problem and works 
towards the answer. Planning is considered backward if it starts with the answer 
needed and then considers how to get that answer. Forward planning is often identified 
with phrases such as, 'let's do [ ] next' or 'where do we go now'. Backward planning 
is often identified with such phrases as 'how can we get that' or 'we need to have [ ]'. 



PL.g: Planning(Goal: Directed or Undirected) 

Planning is considered goal directed if the plan is in pursuit of a goal that has been 
stated explicitly (e.g. by 'we need to do [ ]' or 'we are going to find [ ]'). Planning is 
considered undirected if no goal has been stated explicitly (i.e. by planning what to do 
or how to do it without saying what the desired result is). It is not goal directed 
planning unless the goal has been stated in advance. 

PL«w: Planning(What: Operation, Goal, or Strategy) 

Planning is directed towards deciding on either an operation, a goal to be pursued, 
or a strategy to be used. An operation is narrowly defined as a mathematical operation, 
such as +,-,*,/. A goal is defined as WHAT is desired to be found and a strategy is 
defined as HOW to find what is desired (or by what method). 

EX: ExpIoration(General) 

Exploration is defined as search by constructing trial full or partial solutions. (This can be 
done for fun, in an effort to better understand the problem, or simply as an attempt to find a 
solution when an obvious path is not known.) Exploration can be identified by statements 
such as let's try this' or by (re)constructing a full or partial solution without indication that 
this is believed to be a better way. Simply correcting a mistake is not necessarily trial and 
error. In a trial, there is no expression direct or indirect that this is 'the right way' and there 
may be a statement indicating uncertainty. When a mistake is made, as opposed to a trial, 
there should be direct or indirect evidence that it was believed to be correct at the time. If 
multiple partial solutions are tried at the same point, it would normally be categorized as 
exploration unless there are clear verbalizations that a series of mistakes are being 
corrected. 

Sub-categories of Exploration 

EX*t: Exploration(TriaI and Error) 

Trial and error exploration is unstated or stated but with no evident system of 
organization. To distinguish this from correcting mistakes, look for whether or not 
each partial solution- tried is believed to be correct as tried(mistake) is believed to be 
tentative (trial) or, in the case of multiple partial solutions at the same point, is different 
but v/ith no stated or obvious reason it should be better than the previous partial 
solution. 

EX.s: Exploration(Systematic) 

Systematic exploration attempts to be in accord with a stated system of 
organization, or follows an obvious system of organization (e.g. try all possibilities in 
some kind of order). 

M: MONITORING 

An interval was coded "M" if it included metacognitive activity. Metacognitive activity is 
thinking about thinking. In each subcategory the subject is aware of some aspect of their 
thinking about the problem, the process of problem solving, or the problem solution. 
Comprehension monitoring is looking at the comprehension of the problem, and implies 
awareness of the subject's own comprehension. Error checking is looking at the 
correctness of the solution of the problem, and implies awareness of the correctness of the 
subject's understanding of the problem and the subject's developing solution. 
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Clarifying/questioning is any seeking of further information, and implies awareness of 
missing or ambiguous information. When monitoring is of the interface instead of the 
problem itself, this should be coded by checking the interface category as well. 

Sub-categories of Monitoring: 

M.e: Monitoring(Error Checking: Calc, Concept, Macro, Micro) 

Error checking is looking at a full or partial solution in an attempt to determine if it 
is correct. This can be done as a precaution or because a possible problem has been 
noticed. Error checking is often marked by a statement like 'let's see if this is right' or 
a rhetorical question such as 'is that right?'. Error checking, especially with paired 
subjects, can often have a questioning/clarifying component in parallel as a means to 
seek the possible error. Most error checking is either checking math calculations(calc) 
or checking conceptual understanding(concept). Checking a single step of the problem 
is classified as micro checking as opposed to checking a larger part of the problem 
which is called macro checking. Checking for errors in how the interface is used is also 
possible, and this should be coded as general error checking combined with interface 
interaction. 

M.cm:Monitoring(Comprehension Monitoring) 

comprehension monitoring is defined as any behavior indicating subject's 
awareness of whether subject has or has not correctly comprehended the problem. If 
the subject summarizes part of or the whole problem this is an example of 
comprehension monitoring. Reading activities are excluded as part of the initial attempt 
to comprehend rather than as an attempt to monitor correctness of current 
comprehension. (Underlining was previously included in comprehension monitoring). 
Comments such as 'let's see if I understand this' or 'no, I understood it wrong' 
indicate attempts at comprehension monitoring. A comment like 'did I do this right' 
would not be comprehension monitoring, because it is not aimed at comprehension. 

M.q: Monitoring(Questioning/Clarifying) 

Questioning/clarifying is defined as any attempt to seek information by inquiry or 
discussion. This is possible for individual as well as paired subjects. If the 
information sought is a better comprehension of the problem, it is also a comprehension 
monitoring interchange. It is also possible todiscuss the interface, discuss the 
effectiveness of possible strategies, etc. In pairs this is often indicated when one 
partner asks the other a question. Individuals can ask themselves questions in a verbal 
protocol. Care should be taken to be sure the question indicates the seeking of 
information and to note what type of information is being sought. 



IN: INTERFACE 



Interface interaction is defined as any attention paid to the interface, as opposed to merely 
using it. This often occurs when the subject does not know how to do what they want with 
the interface, or when the interface gives an error message. Interface interaction often 
occurs in parallel with other activity. For example, if the experimenter is asked a question 
about the interface this is coded as questioning/clarifying and interface interaction. This 
coder makes a note whenever the experimenter is involved in any way. Normal use of 
interface without difficulty to solve the problem is NOT in this category. 
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C: CARRYING OUT 



"Carryout" is defined as doing the steps to solve the problem, often in the absence of 
metacognitive activity. (Note taking had previously been in this category.) Carryout can 
occur in concert with other activities. It is often characterized by a lack of statements, other 
than to state what is being done at the moment. Often subjects carrying out a solution 
forget to verbalize entirely, unless reminded. 



TABLE 3. TREATMENT DURATION (EXPRESSED AS TOTAL 
NUMBER OF CODING INTERVALS) AND MEAN PERCENTAGES OF 
THOSE INTERVALS ASSOCIATED WITH TYPES OF COGNITIVE 
ACTIVITY: FOUR TREATMENT GROUPS COMPARED 



IP: 

Individual 
with 
Pen/Paper 



PP: 

Pair 
with 
Pen/Paper 



IT: 

Individual 
with 
Taps 



PT: 

Pair 
with 
Taps 



READING 

s.d. 

PLANNING 

s.d. 

MONITORING 

s.d. 

INTERFACE 

s.d. 

CARRYOUT 

s.d. 



71.46 

12 .40 

0.00 

o.oo 
9.58 

16.93 
NA 
0.00 

45.70 

30.91 



67.78 

15.94 
4.43 
5.96 

48.31 

17 .74 

NA 

0.00 

30.06 

19.91 



28.68 

18.03 

1.86 

3.08 
14.70 

9 .91 
10.71 

10 .07 

52.30 

16.65 



32.00 

12 . 31 
22.71 
12.99 

46.99 

15.76 

9.23 

10.05 

38.40 

16.82 



*DURATION 



;.d. 



11.00 

7 .78 



17.88 

11.42 



40.50 

20 .02 



43.88 

26.05 



* Duration is number of 15 second intervals in the problem solution. 



TABLE 4. TREATMENT MEANS AND RESULTS OF ADJUSTED MANN- 
WHITNEY TESTS OF HYPOTHESIZED TREATMENT EFFECTS 

individual vs. pair pen/paper vs. taps 



READING 50.07 49.89 0.01 69.62 30.34 4.88* 

PLANNING 0.93 13.57 2.47* 2.21 12.29 1.97* 

MONITOR 12.14 47.65 4.35* 28.94 30.84 0.31 

CARRYOUT 49.00 34.23 1.10 37.88 45.35 0.66 

DURATION 25.75 30.88 0.34 14.44 42.19 3.92* 

*P<.05 
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